Global Minima of Overparameterized Neural Networks
نویسندگان
چکیده
Related DatabasesWeb of Science You must be logged in with an active subscription to view this.Article DataHistorySubmitted: 27 December 2019Accepted: 10 February 2021Published online: 13 May 2021Keywordsoptimization, landscape, manifold, eigenvalues the Hessian, overparameterizationAMS Subject Headings68T99, 51M99Publication DataISSN (online): 2577-0187Publisher: Society for Industrial and Applied MathematicsCODEN: sjmdaq
منابع مشابه
Local minima in training of neural networks
Training of a neural network is often formulated as a task of finding a “good” minimum of an error surface the graph of the loss expressed as a function of its weights. Due to the growing popularity of deep learning, the classical problem of studying the error surfaces of neural networks is now in the focus of many researchers. This stems from a long standing question. Given that deep networks ...
متن کاملDeep linear neural networks with arbitrary loss: All local minima are global
We consider deep linear networks with arbitrary differentiable loss. We provide a short and elementary proof of the following fact: all local minima are global minima if each hidden layer is wider than either the input or output layer.
متن کاملLearning behavior and temporary minima of two-layer neural networks
-This paper presents a mathematical analysis of the occurrence of temporary minima during training of a single-output, two-layer neural network, with learning according to the back-propagation algorithm. A new vector decomposition method is introduced, which simplifies the mathematical analysis of learning of neural networks considerably. The analysis shows that temporary minima are inherent to...
متن کاملOn the problem of local minima in recurrent neural networks
Many researchers have recently focused their efforts on devising efficient algorithms, mainly based on optimization schemes, for learning the weights of recurrent neural networks. As in the case of feedforward networks, however, these learning algorithms may get stuck in local minima during gradient descent, thus discovering sub-optimal solutions. This paper analyses the problem of optimal lear...
متن کاملAvoiding Local Minima in Feedforward Neural Networks by Simultaneous Learning
Feedforward neural networks are particularly useful in learning a training dataset without prior knowledge. However, weight adjusting with a gradient descent may result in the local minimum problem. Repeated training with random starting weights is among the popular methods to avoid this problem, but it requires extensive computational time. This paper proposes a simultaneous training method wi...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
ژورنال
عنوان ژورنال: SIAM journal on mathematics of data science
سال: 2021
ISSN: ['2577-0187']
DOI: https://doi.org/10.1137/19m1308943